Sensitive Area Management

ABSTRACT

An unmanned mobile observation device, such as an unmanned aerial vehicle, can be used to acquire observation data for a sensitive area. The observation data can include image data and geolocation data corresponding to a real-world geographic location at which the image data was acquired. The geolocation data can be accurate to five centimeters or less. The observation data can be evaluated to identify any defects within the sensitive area. Such evaluation can include comparison with previously acquired image data for the sensitive area, which can be enables by using the geolocation data to select the corresponding previously acquired image data.

REFERENCE TO RELATED APPLICATIONS

The current application claims the benefit of co-pending U.S.Provisional Application No. 62/675,840, titled “UAV-Based InspectionPlatform for Infrastructure Monitoring,” which was filed on 24 May 2018,and which is hereby incorporated by reference.

TECHNICAL FIELD

The disclosure relates generally to managing an area, and moreparticularly, to managing an area using electromagnetic data acquiredover time.

BACKGROUND ART

Millions of people in the United States drive to work on America'sroads, but poor road infrastructure conditions cost Americans 100billion dollars in wasted fuel and time every year. Maintaining suchinfrastructure is important, but the cost and time needed for humaninspectors to identify defects and judge their severity can beprohibitive. In addition to the cost issue, a small number of humaninspectors in some industries is an issue, with the rail industry forexample, having a notable shortfall. However, infrastructure monitoringis necessary for effective infrastructure maintenance.

With a potential to cover many more miles of additional track in lesstime, the use of unmanned aerial vehicles (UAVs) to locate defects inthe railway industry is currently being tested. In this case, once a UAVhas found a possible defect, a human inspector can be sent to verify thedefect. Similar testing is being done in many areas of infrastructureinspection, such as construction and pipeline inspection. However,current UAV platforms cannot geo-locate defects precisely enough or takeimages of sufficient clarity for this purpose.

Current UAV systems can provide geolocation to an accuracy of a fewinches. However, this error margin is too large for some types ofanalysis, such as matching images of defects with historical image data.Furthermore, the images recorded by UAV systems are often of low qualityor blurred, which makes matching and comparison of image data moreproblematic. For example, in the case of rail infrastructure analysis,smaller defects, such as cracks or missing bolts, may not visible,cannot be identified, or may not be located with sufficient precision.All of this makes it difficult to judge the necessity of humaninfrastructure inspections and repair to the rail through historiccomparisons of image data acquired by UAV systems.

Several approaches to address the geolocation problems have beenproposed. One approach includes an INS/GPS sensor fusion scheme, basedon state-dependent Riccati equation (SDRE) nonlinear filtering, forunmanned aerial vehicle (UAV) localization. This approach improvesaccuracy, but not to the level needed. In addition, georeferenced pointclouds captured at the height of 50 meters can be accurate to 25-40 mm,which is similar to RTK technology widely in use.

Several patents and patent applications describe attempts to resolve thegeolocation problem. For example U.S. Patent Application Publication No.2017/0041763 describes an automated method of determining the locationof an aerial vehicle which uses an aerial platform network. U.S. Pat.No. 9,786,165 describes a system to provide location positioning forlocating a transportation apparatus through a UAV network. U.S. Pat. No.9,678,507 discusses autonomous infrastructure element survey systems andmethods using UAV fleet deployment, which differs considerably as itemploys a fleet of UAVs and is not concerned with the precision oflocation accuracy. U.S. Pat. No. 9,740,200 describes an unmanned aerialvehicle inspection system, in which a UAV flies from one location toanother based on information provided by an operator's device.

However, the accuracy provided by these approaches remains insufficientfor many monitoring applications. Additionally, many of these approachesdo not address infrastructure monitoring, let alone provide an approachto providing an effective solution for such monitoring.

SUMMARY OF THE INVENTION

The inventors recognize that use of commercial drones for monitoring issignificantly limited, particularly with respect to location accuracyand image quality. As a result, effective use of commercial drones todetermine if infrastructure is sound or defective is not possible.

Embodiments described herein provide a management solution that uses anunmanned mobile observation device to acquire observation data, whichincludes highly accurate geolocation data and corresponding image/sensordata, of improvement(s). The observation data allows for comparison ofimage/sensor data acquired over time to determine whether defects,including defects with very small visual footprints, are present orfollow up action(s) are otherwise necessary.

In a more particular embodiment, the unmanned mobile observation deviceis an unmanned aerial vehicle which acquires image/sensor data anddetermines highly accurate location data, which are collectively storedas observation data. The unmanned mobile observation device can becapable of reliably and repeatedly acquiring observation data fromprecise locations. The observation data can be evaluated to identifychanges and/or defects over time.

In an illustrative application, an unmanned mobile observation device isused to acquire observation data for a railroad, such as the railroadtrack, track structure, etc. The observation data can be evaluated by ahuman and/or using data analytics to identify the presence of andlocation of various defects that may be present. For example, theevaluation can indicate that a railroad track has bent since observationdata was previously acquired for the same location.

When a defect/possible defect is identified, the observation data can beused to enable an inspector to locate the defect for in personinspection/verification, a maintenance person to perform maintenance atthe location, follow up imaging to be performed, which can includecloser imaging using an unmanned mobile observation device, and/or thelike. High quality image data can also enable identification ofinfrastructure defects that have a small visual signature, such as afraction of an inch.

Aspects of the invention provide an unmanned mobile observation device,such as an unmanned aerial vehicle, which can be used to acquireobservation data for a sensitive area. The observation data can includeimage data and geolocation data corresponding to a real-world geographiclocation at which the image data was acquired. The geolocation data canbe accurate to five centimeters or less. The observation data can beevaluated to identify any defects within the sensitive area. Suchevaluation can include comparison with previously acquired image datafor the sensitive area, which can be enables by using the geolocationdata to select the corresponding previously acquired image data.

A first aspect of the invention provides a system comprising: anunmanned mobile observation device, the unmanned mobile monitoringdevice including: an imaging component configured to acquire image dataof an improvement being monitored; a navigation component configured tonavigate the unmanned mobile observation device to an image acquisitiongeographic location suitable for acquiring the image data of theimprovement being monitored; and a geolocation component configured todetermine a real-world geographic location of the unmanned mobileobservation device to an accuracy of five centimeters or less when theimaging component acquires the image data, wherein geolocation datacorresponding to the real-world geographic location is stored with imagedata acquired by the imaging component as observation data for theimprovement being monitored.

A second aspect of the invention provides an unmanned aerial vehicle,including: an imaging component configured to acquire image data of anarea being monitored, wherein the imaging component includes: an imagingdevice configured to acquire image data of the area; and a deblurringcomponent configured to remove distortion from the image data acquiredby the imaging device using inertial data of the unmanned aerial vehiclesynchronized with the image data acquisition by the imaging device; anavigation component configured to navigate the unmanned aerial vehicleto an image acquisition geographic location suitable for acquiring theimage data of the area being monitored; and a geolocation componentconfigured to determine a real-world geographic location of the unmannedaerial vehicle to an accuracy of five centimeters or less when theimaging component acquires the image data, wherein geolocation datacorresponding to the real-world geographic location is stored with imagedata acquired by the imaging component as observation data for the areabeing monitored.

A third aspect of the invention provides a system for managing asensitive area, the system comprising: an unmanned aerial vehicle,including: an imaging component configured to acquire image data of anarea being monitored, wherein the imaging component includes: an imagingdevice configured to acquire image data of the area; and a deblurringcomponent configured to remove distortion from the image data acquiredby the imaging device using inertial data of the unmanned aerial vehiclesynchronized with the image data acquisition by the imaging device; anavigation component configured to navigate the unmanned aerial vehicleto an image acquisition geographic location suitable for acquiring theimage data of the area being monitored; a geolocation componentconfigured to determine a real-world geographic location of the unmannedaerial vehicle to an accuracy of one centimeter or less when the imagingcomponent acquires the image data, wherein geolocation datacorresponding to the real-world geographic location is stored with imagedata acquired by the imaging component as observation data for the areabeing monitored; and a monitoring component configured to compare theimage data to previously acquired image data using the geolocation datafor the image data and the previously acquired image data and generatedefect data for each defect found as a result of the image datacomparison; and a sensitive area management system that schedulespersonnel based on the defect data.

Other aspects of the invention provide methods, systems, programproducts, and methods of using and generating each, which include and/orimplement some or all of the actions described herein. The illustrativeaspects of the invention are designed to solve one or more of theproblems herein described and/or one or more other problems notdiscussed.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features of the disclosure will be more readilyunderstood from the following detailed description of the variousaspects of the invention taken in conjunction with the accompanyingdrawings that depict various aspects of the invention.

FIG. 1 shows an illustrative environment for managing a sensitive areaaccording to an embodiment.

FIG. 2 shows an illustrative unmanned aerial vehicle according to anembodiment.

FIG. 3 shows an illustrative conceptual structure of a main body of anunmanned mobile observation device according to an embodiment.

FIG. 4 shows an illustrative processing diagram for generating locationand image data for use in monitoring an improvement according to anembodiment.

FIG. 5 shows an illustrative point spread function, which can be used toremove distortion in the corresponding image data.

FIG. 6 shows illustrative images illustrating distortion removal usingthe point spread function of FIG. 5.

FIG. 7 shows an illustrative unmanned aerial vehicle being used tomonitor a railroad according to an embodiment.

FIG. 8 shows illustrative images of a railroad acquired at differenttimes.

It is noted that the drawings may not be to scale. The drawings areintended to depict only typical aspects of the invention, and thereforeshould not be considered as limiting the scope of the invention. In thedrawings, like numbering represents like elements between the drawings.

DETAILED DESCRIPTION OF THE INVENTION

As indicated above, aspects of the invention provide an unmanned mobileobservation device, such as an unmanned aerial vehicle, which can beused to acquire observation data for a sensitive area. The observationdata can include image data and geolocation data corresponding to areal-world geographic location at which the image data was acquired. Thegeolocation data can be accurate to five centimeters or less. Theobservation data can be evaluated to identify any defects within thesensitive area. Such evaluation can include comparison with previouslyacquired image data for the sensitive area, which can be enables byusing the geolocation data to select the corresponding previouslyacquired image data.

As used herein, “image” refers to any two-dimensional representation ofan area, without regard to how the two-dimensional representation isgenerated. To this extent, an image can be generated based on any typeof electromagnetic radiation, such as ultraviolet, visible, infrared,and/or the like. Furthermore, an image can be generated using atwo-dimensional sensor array or a one-dimensional sensor array thatscans the area. Still further, an image can be generated using othertypes of non-electromagnetic solutions, such as echolocation. Image datarefers to any type of digital representation of the image.

Turning to the drawings, FIG. 1 shows an illustrative environment 10 formanaging a sensitive area according to an embodiment. In general, thesensitive area can include one or more improvements 2. However, it isunderstood that in certain applications described herein, the sensitivearea may not include any improvements 2. As used herein, an improvement2 includes any modification to a geographic area that is subject to wearfrom use, need for maintenance, monitoring for damage, and/or monitoringfor changes. Illustrative improvements include structures, such asrailroads, roads (paved or unpaved), overpasses, bridges, dams,buildings, pipeline, utility poles, wiring and cables, sewers, fencing,walls, gates, etc. Additionally, improvements include paths (e.g.,trails), which may be paved or unpaved, permanent or seasonal, etc. Animprovement can be completed and ready for use or in use, or can beunder construction.

The environment 10 is shown including a sensitive area management system12, which can be configured to manage the use of various devices inmanaging the sensitive area and the improvement(s) 2 located therein.For example, the sensitive area management system 12 can managedeployment and/or operation of one or more inspection devices 14, whichcan be used to acquire data regarding the improvement(s) 2 and determinewhether any maintenance or repairs are required. The inspection devices14 can include, for example, a handheld inspection device which isoperated by a human inspector 4, an automated or semi-automated devicethat can be permanently or temporarily located to monitor theimprovement(s) 2, and/or the like. In addition to the inspection devices14, the sensitive area management system 12 can manage scheduling and/ordeployment of human inspectors 4, who can visually inspect animprovement 2, operate an inspection device 14 to inspect theimprovement(s) 2, and/or the like.

Similarly, the sensitive area management system 12 can manage deploymentand/or operation of one or more maintenance devices 16, which can beused to maintain, repair, etc., the improvement(s) 2. The maintenancedevices 16 can include, for example, a manually operated maintenancedevice which is operated by a human maintenance person 6, an automatedor semi-automated device that can be permanently or temporarily locatedto maintain or repair the improvement(s) 2, and/or the like. In additionto the maintenance devices 16, the sensitive area management system 12can manage scheduling and/or deployment of human maintenance personnel6, who can maintain and/or repair improvement(s) 2, replaceimprovement(s) 2, operate a maintenance device 16 to maintain, repair,and/or replace the improvement(s) 2, and/or the like. While amaintenance person 6 and maintenance device 16 are shown and describedherein, it is understood that the environment 10 can include personneland/or devices that are constructing one or more improvements 2, such asadding an improvement to the area, etc. As part of managing themaintenance devices 16 and maintenance personnel 6, the sensitive areamanagement system 12 can manage ancillary tasks, such as ordering anynecessary parts, tracking wear to predict maintenance, etc.

To reduce the workload of the inspectors 4 and/or maintenance personnel6, and/or provide additional functionality as described herein, thesensitive area management system 12 can further manage deployment and/oroperation of one or more unmanned mobile observation devices 20. As usedherein, an unmanned mobile observation device 20 is any type of unmannedvehicle capable of traveling over land, on or within water, and/or inair, and acquiring image data for use in monitoring one or moreattributes of the area. To this extent, an unmanned mobile observationdevice 20 can comprise a wheeled or tracked vehicle, a boat/ship orsubmarine, or an aircraft. The unmanned mobile observation device 20 canutilize any type of propulsion solution, have any size, have anyoperational range, etc., which are suitable for use of the unmannedmobile observation device 20 in the corresponding environment of thesensitive area including the improvement(s) 2. In a more particularembodiment, the unmanned mobile observation device 20 is capable ofautonomous travel, without control by a human or system operator. In anillustrative embodiment, the image data is generated fromelectromagnetic data. However, it is understood that image datagenerated using other solutions can be acquired.

As illustrated, the unmanned mobile observation device 20 can includevarious components, each of which provides functionality that enablesthe unmanned mobile observation device 20 to be used to effectivelyacquire observation data regarding improvement(s) 2 located within thesensitive area. For example, the unmanned mobile observation device 20can include: a navigation component 22, which is configured to navigatethe unmanned mobile observation device 20 to a desired location withrespect to an improvement 2 to be observed; an imaging component 24,which is configured to acquire image data of the improvement 2; ageolocation component 26, which is configured to precisely identify areal-world geographic location of the unmanned mobile observation device20 when the imaging component 24 acquires the image data; and amonitoring component 28, which enables the unmanned mobile observationdevice 20 to execute a monitoring process in an autonomous orsemi-autonomous manner. Further details and illustrative attributes ofthe unmanned mobile observation device 20 and the correspondingcomponents 22, 24, 26, 28 are described herein. However, it isunderstood that this description is not intended to limit the unmannedmobile observation device 20 to the precise embodiments shown anddescribed, but is intended to illustrate various features of an unmannedmobile observation device described herein.

In a more particular illustrative embodiment, the unmanned mobileobservation device 20 is an unmanned aerial vehicle (UAV). To thisextent, FIG. 2 shows an illustrative UAV 20A according to an embodiment.As illustrated, the UAV 20A includes a main body 30, which is liftedinto flight by a propulsion system including four propellers 32A-32D.The propulsion system can be powered using any type of engine or motor,which can use any type of power source(s), such as a fuel, batteries,etc. Additionally, the UAV 20A includes a pair of skids 34A, 34B, whichenable the UAV 20A to be landed without damaging the main body 30 andthe corresponding components located therein. It is understood that theconfiguration of UAV 20A is only illustrative of various possibleconfigurations of UAVs possible for use in embodiments of unmannedmobile observation devices described herein.

FIG. 3 shows an illustrative conceptual structure of a main body 30A ofan unmanned mobile observation device 20 (FIG. 1) according to anembodiment. In general, the main body 30A can house a control unit 40,which comprises various devices for safely navigating the unmannedmobile observation device 20 and acquiring observation data, which canbe used in evaluating an improvement 2 (FIG. 1) as described herein. Ingeneral, the control unit 40 can comprise a processing component 48A, astorage component 48B, and a power component 48C, which providefunctionality to one or more of the other components included in thecontrol unit 40.

For example, the processing component 48A can include one or moregeneral purpose and/or single purpose (application specific) processors,each of which is configured to process data according to instructions(e.g., as defined by circuitry, program code, such as a monitoringprogram 48E, executed by a processor, a field-programmable gate array,and/or the like), which can result in the processing component 48Aobtaining data from the storage component 48B and/or another component,providing transformed data to the storage component 48B and/or anothercomponent, etc. To this extent, the processing component 48A can includeone or more processors used by multiple components described hereinand/or one or more processors used only by a single component describedherein. A processor can be used to perform the same processing taskand/or be used to perform different processing tasks (e.g., by executingdifferent program code) by one or more of the components describedherein.

The storage component 48B can include, for example, a storage hierarchy,which includes multiple types of data storage solutions. For example,the storage component 48B can include random access memory (RAM), flashmemory, disk storage (e.g., solid-state, magnetic, optical, and/or thelike), etc. In an embodiment, the storage component 48B includessufficient disk storage space to store all observation data 49A acquiredof improvement(s) 2 while performing a monitoring task described herein.Alternatively, the storage component 48B can store only a portion of theobservation data 49A, and the control unit 40 can be configured totransmit observation data 49A and/or data derived therefrom, such asdefect data 49B, to another computer system located apart from theunmanned mobile observation device 20 while performing a monitoring taskdescribed herein.

The power component 48C can comprise, for example, a set of batteries,which are configured to provide sufficient power to operate allcomponents of the control unit 40 for a sufficient time to perform amonitoring task described herein. Additionally, the power component 48Ccan include one or more devices configured to recharge the batteryduring use of the unmanned mobile observation device 20, e.g., usingelectricity generated during operation of the unmanned mobileobservation device 20 (e.g., by an alternator), generating electricityfrom an ambient operating condition (e.g., solar cells), etc.

As discussed herein, the control unit 40 of the unmanned mobileobservation device 20 can include a navigation component 22, whichenables the unmanned mobile observation device 20 to move within itsoperating area to a target location, e.g., to observe an improvement 2.Using the UAV 20A (FIG. 2) as an illustrative unmanned mobileobservation device 20, the navigation component 22 can include: anelectronic speed controller 42A, a flight controller 42B, a collisionavoidance sensor 42C, a flight camera 42D, a global positioning system(GPS) sensor 42E, and a GPS antenna 42F.

During operation of the navigation component 22, the electronic speedcontroller 42A can control the propulsion system (e.g., the propellers32A-32D shown in FIG. 2) to control the speed at which the unmannedmobile observation device 20 is moving. The flight controller 42B canoperate in tandem with the collision avoidance sensor 42C to adjust thespeed and/or direction of travel of the unmanned mobile observationdevice 20 in order to safely navigate within the operating environment.

The flight controller 42B can further acquire and process data (e.g.,using the processing component 48A) from a flight camera 42D and/or aGPS sensor 42E in order to navigate to a particular location. Forexample, the GPS sensor 42E can process GPS signal data acquired by theGPS antenna 42F to determine a location of the unmanned mobileobservation device 20. While a GPS sensor 42E and GPS antenna 42F areshown, it is understood that this is only illustrative of the globalnavigation satellite systems (GNSS) which can be utilized in embodimentsdescribed herein. The flight camera 42D can acquire image data of theoperating environment to identify known features and/or the location ofobstacles to be avoided. Additionally, the flight camera 42D can acquireimage data that can be transmitted to an external navigation system,which is used by a human operator to monitor the movement of theunmanned mobile observation device 20 and/or navigate the unmannedmobile observation device 20 during a manual navigation mode.

The control unit 40 of the unmanned mobile observation device 20 alsocan include an imaging component 24, which is configured to acquireimage data of the improvement 2. For example, the imaging component 24is shown including a camera and gimbal 44A. The gimbal can mount thecamera to the main body 30A and can enable the camera to be rotatedabout one or more axes. The camera can comprise any type of imagingdevice that acquires image data of a sufficient quality to enableanalysis of the improvement 2 as described herein. For example, for manyapplications, a 100 mega pixel camera resolution can be utilized.However, it is understood that higher or lower resolutions can besuitable for use in various applications. In an embodiment, the camerais a visible light-based imaging device. However, it is understood thatthe imaging component 24 can include one or more sensors 44B thatgenerate image data from other spectra, such as ultraviolet, infrared,multiple spectra, etc. Furthermore, the imaging component 24 can includea sensor 44B that generates image data from another sensing approach,such as using three-dimensional scanning (e.g., a structured light 3Dscanner), light detection and ranging (LIDAR), radar, sonar, and/or thelike.

The imaging component 24 also can include one or more componentsconfigured to improve a quality of the image data. For example, theimaging component 24 can include a deblurring component 44C, which canbe configured to remove distortion from the image data. Additionally,the imaging component 24 can include a high resolution (high res)component 44D, which can be configured to generate image data having ahigher resolution than the raw image data by combining two or moreimages of an area. In an embodiment, one or both of the deblurringcomponent 44C and the high resolution component 44D is implemented asprogram code executed by the processing component 48A.

Furthermore, the control unit 40 is shown including a geolocationcomponent 26, which is configured to precisely determine a real-worldgeographic location of the unmanned mobile observation device 20. Tothis extent, the geolocation component 26 includes various sensors whichprovide data which can be used to further refine the accuracy of thelocation of the unmanned mobile observation device 20 as determined fromthe GPS data. Additionally, the geolocation component 26 can acquiredata used to provide positioning for a period of time when GPS data maynot be available. For example, the geolocation component 26 is shownincluding: an accelerometer 46A, which can acquire proper accelerationdata for the unmanned mobile observation device 20; a gyroscope 46B,which can acquire data regarding orientation and angular velocity ofunmanned mobile observation device 20; an altimeter 46C, which canacquire data corresponding to an altitude of the unmanned mobileobservation device 20; and a magnetometer 46D, which can acquire dataregarding a magnetic field, such as the ambient magnetic field withinwhich the unmanned mobile observation device 20 is located. In anembodiment, the various sensors 46A-46D are implemented in an inertialmanagement unit (IMU). However, it is understood that the particularimplementation and combination of sensors 46A-46D is only illustrativeof various sensor configurations that can be utilized. Additionally, thegeolocation component 26 can include a ground-based correction sensor46E, which can receive location signals from a nearby ground-basedlocation.

The control unit 40 also can include a monitoring component 28, whichenables the unmanned mobile observation device 20 to execute amonitoring process in an autonomous or semi-autonomous manner. As partof executing a monitoring process, the control unit 40 may receive andprocess, e.g., as defined by the monitoring program 48E, a route whichdefines the location(s) to which the unmanned mobile observation device20 should navigate and acquire observation data 49A of a sensitive area,e.g., one or more improvements located therein. Data defining such aroute can be stored in the storage component 48B and processed by theprocessing component 48A to direct the navigation component 22 toautonomously navigate to the corresponding location(s) on the route andacquire observation data 49A of the improvement(s) 2.

Additionally, the control unit 40 can include one or more microelectro-mechanical systems (MEMS) sensors 48D as part of the monitoringcomponent 28. In an embodiment, the MEMS sensors 48D can include one ormore accelerometers, one or more gyroscopes, and/or the like. However,it is understood that these sensors are only illustrative. As describedherein, image data acquired by the imaging component 24 may requirefurther processing to be suitable for data analysis defines as part ofthe monitoring process. The MEMS sensor 48D can acquire data that can beused to deblur and/or enhance the raw image data acquired by the imagingcomponent 24. The data acquired by the MEMS sensor 48D can be stored asobservation data 49A in the storage component 48D and/or processed bythe processing component 48A to refine the raw image data for furtheranalysis.

While not shown in FIG. 3, it is understood that an unmanned mobileobservation device 20, e.g., the main body and/or control unit thereof,can include one or more additional components that enable wirelesscommunications with one or more external systems. To this extent, thecontrol unit 40 can further include one or more transceivers and/orcorresponding antennae that enable the unmanned mobile observationdevice 20 to communicate with an external system using any wirelesscommunications solution. Illustrative wireless communications solutionsinclude a wireless radio link (e.g., direct or indirect), communicationsusing a cellular network (which can utilized a public telephonenetwork), satellite communications, and/or the like. Additionally, thecontrol unit 40 can include one or more communications devices forcommunicating over a wired communications link, e.g., to transfer datafrom the unmanned mobile observation device 20 after returning fromacquiring observation data for improvement(s) 2. To this extent, it isunderstood that many modifications and variations of the componentsdescribed herein can be implemented in embodiments of an unmanned mobileobservation device 20 described herein.

As described herein, an embodiment of the unmanned mobile observationdevice 20 can include precise geolocation information in the observationdata 49A corresponding to a location at which image data of animprovement 2 was acquired by the imaging component 24 (e.g., by animaging device or other sensor). Furthermore, the unmanned mobileobservation device 20 can include precise geolocation data in theobservation data 49A for image data of the improvement 2 acquired atdifferent times, e.g., during different traversals of a monitoringroute. Inclusion of precise geolocation data for the image data canenable consistent utilization of the observation data 49A forcomparisons of attributes of the improvement 2 at different times aspart of monitoring the improvement 2 for required maintenance, repair,replacement, and/or the like.

FIG. 4 shows an illustrative processing diagram for generating locationand image data for use in monitoring an improvement according to anembodiment. The process can be performed on the unmanned mobileobservation device 20 (FIG. 1) and/or portions of the process can beperformed by another system other than the unmanned mobile observationdevice 20, such as the sensitive area management system 12 (FIG. 1), aninspection device 14 (FIG. 1), and/or the like. In an embodiment, atleast a portion of the actions included in the process are defined bythe monitoring program 48E (FIG. 3), which is executed by the processingcomponent 48A (FIG. 3) to perform the actions described herein. As anillustrative embodiment, the process is described as being entirelyperformed by the unmanned mobile observation device 20. However, it isunderstood that this is only illustrative and at least some of the imageprocessing, location refinement, and/or image analysis described hereincan be performed and/or replicated by another system, such as thesensitive area management system 12 (FIG. 1), using the image andlocation data acquired by the unmanned mobile observation device 20.

In action 50, the processing component 48A can receive GPS signal datafrom a GPS satellite, which was acquired by the GPS sensor 42E (FIG. 3)using any of various GPS modes, such as a differential GPS mode. Theprocessing component 48A can perform real-time kinematic (RTK)positioning using measurements of the phase of the GPS signal's carrierwave to improve an accuracy of the GPS position data. The accuracy ofcurrent RTK is limited by distance dependent errors from orbit,ionosphere, and troposphere, as well as other local effects.

To further improve the accuracy of the position data, in action 52, theprocessing component 48A can process correction signal data received bya correction sensor 46E (FIG. 3) from an inspection device 14A. Thecorrection signal data can include data regarding coordinates and acorrection. The inspection device 14A can comprise a fixed or mobileinspection device 14A, which can very quickly acquire GPS signals,compute corrections, and transmit the corrections to the nearby unmannedmobile observation device 20. In an embodiment, the unmanned mobileobservation device 20 can be located up to approximately twentykilometers (12.4 miles) from the inspection device 14A. The processingcomponent 48A can use the correction signal data to mitigate spatialcorrelation errors and speed up the process of resolving ambiguity inprecise positioning. This combination of RTK positioning with correctioncan achieve geolocation accuracy of 2-5 centimeters (approximately 0.8to 1.9 inches). While a single inspection device 14A is shown providinga correction signal, it is understood that an unmanned mobileobservation device 20 can receive and process correction signal datafrom any number of one or more inspection devices 14A, which can includeany combination of fixed and/or mobile inspection devices 14A. In anembodiment, the inspection device 14A comprises a vehicle.

In action 54, the processing component 48A can further improve anaccuracy of the location data using inertial measurement data acquiredby the geolocation component 26 (FIG. 3). The inertial measurement datacan comprise a combination of sensor data acquired by multiple types ofsensors, such as some or all of the sensors described in conjunctionwith the geolocation component 26 shown in FIG. 3, which the processingcomponent 48A can use to further refine an accuracy of the location datato attain the most reliable and precise location data available.Additionally, the processing component 48A can use the inertialmeasurement data to provide location guidance when GPS signal data isnot available. By using a combination of the RTK positioning withcorrection and inertial measurement data, the processing component 48Acan determine a location to an accuracy of less than 1 centimeter (e.g.,approximately 0.4 inches).

The processing component 48A also can improve quality of the image dataacquired by the imaging component 24 (FIG. 1). For example, in manysituations, the acquired image data can include distortion (e.g., blur)due to motion of the unmanned mobile observation device 20 duringacquisition of the image data. In an embodiment, an imaging device canimplement image stabilization and vibration reduction techniques thatconsumer cameras implement eliminate many camera shake effects. Forexample, dual axis rate sensors in the imaging device can measure camerapitch and yaw and actuators can drive an optical element to keep theimage motionless on the imaging device. However, as imagery acquired bya UAV is most affected by roll and pitch of the airframe relative to thedesired flight path, steered optical correction of roll is not asstraightforward as pitch and yaw correction.

To this extent, in action 56, the processing component 48A can removedistortion from the image data (e.g., as directed by the deblurringcomponent 44C). In an embodiment, the processing component 48A removesdistortion using data regarding a motion trajectory, such as a pointspread function (PSF), of the imaging device (e.g., the camera) relativeto the object space during the acquisition time of the image data. Ifthe PSF is perfectly known for an image, the intra-frame motion blur canbe completely removed, less the high frequency information permanentlylost.

In an embodiment, the processing component 48A can implement adistortion removal technique that uses synchronized inertial data todevelop a PSF that is used to remove distortion from the image datausing a set of deconvolution solutions. For example, the MEMS sensor 48D(FIG. 3) can acquire data regarding roll and pitch of the unmannedmobile observation device 20 at a sampling rate of approximately 200Hertz. For an imaging device that generates images at up to 30 framesper second, a frame time will be approximately 33 milliseconds. As aresult, the MEMS sensor 48D will acquire about seven discretemeasurements of the roll and pitch angular rates during the frame time.However, it is understood that the sampling and imaging rates are onlyillustrative.

From the discrete measurements of the roll and pitch, the processingcomponent 48A can generate a PSF for an image. To this extent, FIG. 5shows an illustrative point spread function 56A, which can be used toremove distortion in the corresponding image data. As illustrated, thediscrete measurements are converted into a continuous PSF 56A. The PSF56A shows a large circular blur that is difficult to deblur usingconventional methods. Distortion removal can use the fact that energy isconserved in the blurring process. As a result, the total incidentradiation in the image data acquired during the frame time is the samein a blurred image as it would be if there were no blur. Using this factas well as the PSF 56A enables the processing component 48A to removethe majority of the motion-based blur from the image.

FIG. 6 shows illustrative images 56B-56D illustrating distortion removalusing the PSF 56A of FIG. 5. In particular, image 56B comprises a rawimage that includes distortion due to the circular motion that occurredduring the frame time as estimated by the PSF 56A. Image 56C correspondsto the raw image after distortion has been removed using the PSF 56A,which can be generated as described herein. Image 56D is an image thatdoes not include any distortion. As illustrated by the image 56C,numerous features of the image are enhanced as a result of thedistortion removal process described herein.

Returning to FIG. 4, in action 58, the processing component 48A canenhance a resolution of image data acquired of an improvement 2 usingany solution. For example, the processing component 48A can use asuper-resolution process to develop a set of higher resolution imagesfrom lower resolution images. In this process, the processing component48A can create one upsized image of improved resolution usinginformation from several different images. For example, resolution ofthe upsized image can be increased from that of the lower resolutionimages by the square root of the number of usable lower resolutionimages used. To this extent, by using the information of two lowerresolution images, the resolution can be increased by approximatelyforty percent.

An increase in resolution can be beneficial for numerous reasons. Forexample, in infrastructure image recording operations, the higherresolution image data can be used to identify the presence or absence ofpotential issues having a smaller visual footprint, such as identifyingmissing bolts on railway lines, examining rail cross tie conditions orskew, examining gas pipelines for faults, etc. Using a process describedherein, as well as common image processing algorithms, such asvalidation, the processing component 48A can create high resolutionreference images of an improvement 2. This process can further improveresolution of features to the 3 to 5 pixel level or lower. In anembodiment, the process enables images (e.g., previously acquired andnewly acquired high resolution reference images) to be accuratelyaligned within 1-2 pixels of the image data for comparison. As a result,comparison of the respective images can identify changes (e.g., defects)that result in only minor differences in the image data.

In action 60, the processing component 48A can store the geolocation andimage data as observation data 49A (FIG. 3) in the storage component 48B(FIG. 3) for later retrieval using any data storage structure, such as adatabase. Additionally, the processing component 48A can compare imagedata, e.g., such as image data of the same improvement 2 acquired atdifferent times, to identify any features that may have moved, aremissing, have changed, and/or the like. As part of the image comparison,the processing component 48A can implement any of various imageprocessing algorithms to compare new images or data to historic imagesor data.

To this extent, illustrative image processing algorithms include: imagesubtraction; image cross correlation; image feature recognition (e.g.,tie plate bolt hole patterns or ballast quality feature recognition byusing histogram of gradient, Harr wavelet, Tensor flow, SIFT, or deeplearning); and/or any other image processing algorithms which willprovide the required output relevant in each case. The processingcomponent 48A can identify the desired results from the observation data49A through the image processing and analytics, e.g., through smartregistration of images, validation rules, image processing enhancementand differentials, and/or the like. Due to significant improvement ingeolocation and image processing down to a pixel level resolution, thecombined output from the processing component 48A can quickly andefficiently identify and log defects that result in visible changes tothe appearance of the improvement 2. The processing component 48A canstore the results of the analysis as defect data 49B, which can includeidentification of a defect found and/or operating status of animprovement 2, as well as the associated geolocation informationassociated with the defect.

As described herein, the unmanned mobile observation device 20 can beutilized to acquire observation data and/or monitor various types ofimprovements. In a more particular embodiment, the unmanned mobileobservation device 20 described herein can be utilized to monitor acondition of infrastructure, such as an improvement relating totransportation, communication systems, power, water, sewage, and/or thelike.

To this extent, FIG. 7 shows an illustrative unmanned mobile observationdevice, such as the UAV 20A shown in FIG. 2, being used to monitor arailroad 2A according to an embodiment. As described herein, the UAV 20Acan use GPS signals received from a GPS satellite and/or correctionsignals received from an inspection device 14A to determine a geographiclocation of the UAV 20A with a high degree of accuracy. When properlypositioned, the UAV 20A can acquire image/sensor data regarding therailroad 2A, which can be processed as described herein in order todetermine whether the railroad 2A is in need of any maintenance, furtherinspection, and/or the like.

The UAV 20A can store the image/sensor data with the location data asobservation data 49A (FIG. 4) to enable later access and use, e.g., tocompare differences in the railroad over time. For example, FIG. 8 showsillustrative images 62A, 62B of a railroad 2A acquired at differenttimes. The UAV 20A can use the geolocation data to select an historicimage 62A for comparison with a newly acquired image 62B. Data regardingdifferences in the two images, as illustrated by possible defects 64A,64B, can be stored as defect data 49B for follow up. The defect data 49Bcan include the image data, which can be annotated to highlight thedifference(s), location data corresponding to the location at which theimage was taken, identification of the defect, and/or the like. Suchfollow up can include deployment of a human inspector 4 and/or one ormore inspection devices 14 for verification, deployment of a maintenanceperson 6 and/or one or more maintenance devices 16 for repair, removalof the railroad 2A from service, modification of the operatingconditions for trains using the railroad 2A (e.g., a temporary slowingof the speed limit), and/or the like. For example, a rail inspector canbe able to determine what changes have occurred and determine if railstructure defects, such as missing bolts, ballast problems, buckledrails, etc., are present.

As described herein, the unmanned mobile observation device, such as theUAV 20A, can be used to repeatedly acquire image/sensor data of animprovement for use in monitoring the improvement over time. While theunmanned mobile observation device can be manually navigated, theunmanned mobile observation device can be more accurately and repeatedlylocated when navigating autonomously. To this extent, the unmannedmobile observation device can be programmed to repeatedly follow apredetermined navigation (e.g., flight) path, a defined line, and/or thelike, and acquire image data at the same location(s) along the path.Repeated use of the same path and/or image location(s) can furtherimprove the results of comparisons with historic images/data as bothsets of images would be acquired from the same path.

It is understood that other solutions for flight path planning can beutilized without deviating from the spirit of this invention. Suchsolutions include, but are not limited to: following a line such as arailway line, road, fence, wall, pipeline, etc.; following a programmedpath around or inside of a structure, such as a building, house, bridge,and so on; and/or the like. It is understood that the use of anyprogrammed flight path can provide the UAV a similar vantage point asthat used historically from which to acquire image/sensor data forcomparing to historic image/sensor data.

While managing a railroad is one possible application, it is understoodthat a system described herein can be used as part of managing any ofvarious types of sensitive areas with improvement(s). For example, a UAVdescribed herein can be used to perform other types of infrastructureinspections, e.g., to acquire observation data of any infrastructure,including but not limited to: highway bridges and overpasses (from aboveand/or below), buildings, roads, etc. By comparison with historicobservation data, the system can determine when human inspection and/orrepair is required.

Similar to railroad management and inspection, the UAV can be used tomanage other types of “lines,” such as the line of a pipeline, powerwires, a road, fencing, etc., by following the line to check forpossible defects which could be subsequently verified by a humaninspector. Where heat may indicate a problem (e.g., leaks, overheating,etc.), the UAV can incorporate infrared imaging. Historic observationdata can be used for comparison as described herein.

Apart from monitoring improvements, an unmanned mobile observationdevice described herein can be used to monitor an area, e.g., forsecurity and/or safety purposes. For example, a UAV described herein canbe used to monitor environmental areas for problems, including but notlimited to: forest fires, avalanches, deforestation, volcanic activity,floods, erosion, sinkholes, landslides, changing coastlines, and so on.In this case, the UAV can repeatedly acquire observation data that canbe compared with historic observation data for changes indicative ofpotential problems. Similarly, a UAV described herein can repeatedlyacquire observation data of a landscape, such as a border area, and noteany changes, which may indicate border crossings, broken walls,dangerous or criminal activities, illegal border crossings, intrusions,and so on. Such monitoring can be performed on security-relatedimprovements, such as walls, fences, gates, etc., as well as sensitivebuildings such as nuclear facilities or similar, and so on. Stillfurther, an unmanned mobile observation device described herein can beused to periodically acquire observation data of a construction site,and through image comparison, determine the amount of progress made overtime at the site.

Various applications can use non-aerial unmanned mobile observationdevices, such as robots or underwater drones, which can includecomponents similar to those described herein to geolocate and processimages when performing infrastructure inspections, such as of underwaterpipes, cables, etc., or by land-based autonomous vehicles for a widevariety of inspections.

As used herein, unless otherwise noted, the term “set” means one or more(i.e., at least one) and the phrase “any solution” means any now knownor later developed solution. The singular forms “a,” “an,” and “the”include the plural forms as well, unless the context clearly indicatesotherwise. Additionally, the terms “comprises,” “includes,” “has,” andrelated forms of each, when used in this specification, specify thepresence of stated features, but do not preclude the presence oraddition of one or more other features and/or groups thereof.

The foregoing description of various aspects of the invention has beenpresented for purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formdisclosed, and obviously, many modifications and variations arepossible. Such modifications and variations that may be apparent to anindividual in the art are included within the scope of the invention asdefined by the accompanying claims.

What is claimed is:
 1. A system comprising: an unmanned mobileobservation device, the unmanned mobile monitoring device including: animaging component configured to acquire image data of an improvementbeing monitored; a navigation component configured to navigate theunmanned mobile observation device to an image acquisition geographiclocation suitable for acquiring the image data of the improvement beingmonitored; and a geolocation component configured to determine areal-world geographic location of the unmanned mobile observation deviceto an accuracy of five centimeters or less when the imaging componentacquires the image data, wherein geolocation data corresponding to thereal-world geographic location is stored with image data acquired by theimaging component as observation data for the improvement beingmonitored.
 2. The system of claim 1, wherein the image acquisitiongeographic location corresponds to a geographic location at which aprevious image of the improvement was acquired.
 3. The system of claim1, wherein the unmanned mobile observation device is an unmanned aerialvehicle.
 4. The system of claim 3, wherein the real-world geographiclocation includes an altitude of the unmanned mobile observation device.5. The system of claim 1, wherein the navigation component includes aglobal navigation satellite system (GNSS) sensor configured to receiveGNSS signals transmitted by GNSS satellites, and wherein the geolocationcomponent includes a ground-based correction sensor configured toreceive location signals from a ground-based location, wherein thegeolocation component determines the real-world geographic location ofthe unmanned mobile observation device to the accuracy of fivecentimeters or less using data regarding the GNSS signals received bythe GNSS sensor and data regarding the location signals received by theground-based correction sensor.
 6. The system of claim 5, wherein thegeolocation component further includes an inertial management component,wherein the inertial management component uses data from a plurality ofsensors to further refine the real-world geographic location to anaccuracy of one centimeter or less.
 7. The system of claim 1, whereinthe imaging component includes: a visible imaging device configured toacquire image data based on visible light; and a deblurring componentconfigured to remove distortion from the image data acquired by thevisible imaging device.
 8. The system of claim 7, wherein the deblurringcomponent uses inertial data of the unmanned mobile observation devicesynchronized with the image data acquisition by the visible imagingdevice to generate a point spread function.
 9. The system of claim 8,wherein a set of deconvolution solutions use the point spread functionto remove the distortion.
 10. The system of claim 8, wherein theinertial data comprises roll and pitch rate data acquired by amulti-axis sensor.
 11. The system of claim 7, wherein the imagingcomponent further includes a high resolution component configured togenerate a high resolution image by combining a plurality of imagesacquired by the visible imaging device, wherein the high resolutionimage has a resolution at least forty percent higher than the resolutionof the plurality of images acquired by the visible imaging device. 12.The system of claim 1, the unmanned mobile observation device furtherincluding a monitoring component configured to autonomously route theunmanned mobile observation device along a pre-determined route toacquire observation data for the improvement.
 13. The system of claim 1,the unmanned mobile observation device further including a monitoringcomponent configured to compare the image data to previously acquiredimage data using the geolocation data for the image data and thepreviously acquired image data and generate defect data for each defectfound as a result of the image data comparison.
 14. The system of claim13, further comprising a sensitive area management system that schedulesat least one of: a set of inspectors or a set of maintenance personnelbased on the defect data.
 15. An unmanned aerial vehicle, including: animaging component configured to acquire image data of an area beingmonitored, wherein the imaging component includes: an imaging deviceconfigured to acquire image data of the area; and a deblurring componentconfigured to remove distortion from the image data acquired by theimaging device using inertial data of the unmanned aerial vehiclesynchronized with the image data acquisition by the imaging device; anavigation component configured to navigate the unmanned aerial vehicleto an image acquisition geographic location suitable for acquiring theimage data of the area being monitored; and a geolocation componentconfigured to determine a real-world geographic location of the unmannedaerial vehicle to an accuracy of five centimeters or less when theimaging component acquires the image data, wherein geolocation datacorresponding to the real-world geographic location is stored with imagedata acquired by the imaging component as observation data for the areabeing monitored.
 16. The unmanned aerial vehicle of claim 15, whereinthe geolocation component further includes an inertial managementcomponent, wherein the inertial management component uses data from aplurality of sensors to further refine the real-world geographiclocation to an accuracy of one centimeter or less.
 17. The unmannedaerial vehicle of claim 15, further comprising a monitoring componentconfigured to compare the image data to previously acquired image datausing the geolocation data for the image data and the previouslyacquired image data and generate defect data for each defect found as aresult of the image data comparison.
 18. The unmanned aerial vehicle ofclaim 15, further comprising a monitoring component configured toautonomously route the unmanned mobile observation device along apre-determined route to acquire observation data for the area.
 19. Asystem for managing a sensitive area, the system comprising: an unmannedaerial vehicle, including: an imaging component configured to acquireimage data of an area being monitored, wherein the imaging componentincludes: an imaging device configured to acquire image data of thearea; and a deblurring component configured to remove distortion fromthe image data acquired by the imaging device using inertial data of theunmanned aerial vehicle synchronized with the image data acquisition bythe imaging device; a navigation component configured to navigate theunmanned aerial vehicle to an image acquisition geographic locationsuitable for acquiring the image data of the area being monitored; ageolocation component configured to determine a real-world geographiclocation of the unmanned aerial vehicle to an accuracy of one centimeteror less when the imaging component acquires the image data, whereingeolocation data corresponding to the real-world geographic location isstored with image data acquired by the imaging component as observationdata for the area being monitored; and a monitoring component configuredto compare the image data to previously acquired image data using thegeolocation data for the image data and the previously acquired imagedata and generate defect data for each defect found as a result of theimage data comparison; and a sensitive area management system thatschedules personnel based on the defect data.
 20. The system of claim19, wherein the area includes a set of improvements, wherein thepersonnel include at least one of: a human inspector or a maintenanceperson.